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Abstract. A series of experiments measured the audiovisual stimulus onset asynchrony 
(SOA AV ), yielding facilitative multisensory integration. We evaluated (1) the range of SOA AV 
over which facilitation occurred when unisensory stimuli were weak; (2) whether the range 
of SOA AV producing facilitation supported the hypothesis that physiological simultaneity 
of unisensory activity governs multisensory facilitation; and (3) whether AV multisensory 
facilitation depended on relative stimulus intensity. We compared response-time distributions 
to unisensory auditory (A) and visual (V) stimuli with those to AV stimuli over a wide range 
(300 and 20 ms increments) of SOA AV , across four conditions of varying stimulus intensity. In 
condition 1, the intensity of unisensory stimuli was adjusted such that d' « 2. In condition 2, 
V stimulus intensity was increased (c/' > 4), while A stimulus intensity was as in condition 1. 
In condition 3, A stimulus intensity was increased (d' > 4) while V stimulus intensity was 
as in condition 1. In condition 4, both A and V stimulus intensities were increased to 
clearly suprathreshold levels (d' > 4). Across all conditions of stimulus intensity, significant 
multisensory facilitation occurred exclusively for simultaneously presented A and V stimuli. In 
addition, facilitation increased as stimulus intensity increased, in disagreement with inverse 
effectiveness. These results indicate that the requirements for facilitative multisensory 
integration include both physical and physiological simultaneity. 

Keywords: multisensory integration, neural coactivation, inverse effectiveness, race model, simultaneity, reaction 
time, d'. 

1 Introduction 

1.1 Multisensory integration 

Meredith ( 2002 ) identified two classes of multisensory convergence: areal and neuronal. Areal conver- 
gence occurs when unisensory neurons from different modalities merely coexist within a brain region 
but do not interact. Neuronal convergence occurs when unisensory neurons from two or more modali- 
ties make synaptic contact onto recipient "multisensory" neurons. Multisensory integration occurs 
when the response of multisensory neurons to convergent unisensory input differs qualitatively and 
quantitatively from that elicited by the individual unisensory inputs alone (Calvert, 2001 ). 

1 .2 The redundant signals effect 

A well-known behavioral manifestation of facilitative multisensory integration is the decrease in 
response time (RT) to pairings of unisensory stimuli presented over multiple sensory channels, where 
RT to the multisensory combination is faster than to either unisensory signal alone. This enhancement in 
the speed of processing has been termed the "redundant signals effect," or RSE (Miller, 1982 ). The RSE 
is not an exclusively multisensory phenomenon because it also occurs when the redundant signals occur 
within a single sensory modality (Iacoboni & Zaidel, 2003 ; Miller, 1986 ; Miniussi, Girelli, & Marzi, 
1998 ; Molholm et al, 2002 ; Mordkoff & Yantis, 1991 ; Murray, Foxe, Higgins, Javitt, & Schroeder, 
2001 ; Supek et al., 1999 ), but this paradigm has been widely employed in multisensory research. 
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Miller ( 1982 ) compared two models that could potentially account for the RSE: separate acti- 
vation or "race" models and neural coactivation models. Race models assume that each unisensory 
(redundant) signal is processed independently, such that on each trial the channel that processes the 
stimulus most quickly, and thus initiates the overt response, "wins" the race. According to race models, 
redundancy gains can result solely from statistical probability summation and no neural interaction 
between the activated sensory channels is required (although such interactions are not ruled out). In 
contrast, neural coactivation models posit that signals from each unisensory channel interact in order 
to initiate the response. Coactivation across the two channels accumulates until a response criterion is 
reached, which can occur before the same criterion is reached by activation within either individual 
channel. 

Miller ( 1982 ) derived a mathematical inequality describing the race model that specifies an upper 
limit on the cumulative probability (CP) of obtaining speeded RT to redundant stimuli. Miller's ine- 
quality asserts that the CP of obtaining the fastest responses to redundant signals must be less than or 
equal to the CP of obtaining the fastest responses to individual stimuli. Specifically, the race model 
states that for pairs of stimuli, i.e., auditory (A) and visual (V), at a given AV stimulus onset asyn- 
chrony (SOA AV ), at a given response latency if), that 



where V is delayed relative to A. CP(RT < t | AV) is the CP of obtaining an RT faster than time (t) 
in response to the presentation of the A and V stimuli. This CP must be less than or equal to the 
sum of the CPs of obtaining RTs faster than time in response to the individual unisensory stimuli, 
CP(RT < /-SOA AV |A) + CP(RT < T\V) or CP(RT < t\A) + CP(RT < ;-SOA AV |V). Violations of 
Miller's inequality signify that probability summation cannot account for decreased RTs in response to 
redundant signals and implies that neural coactivation has occurred. The evidence that neural coactiva- 
tion is a causal mechanism for the AV RSE comes from numerous behavioral and electrophysiological 
studies (Giard & Peronnet, 1999 ; Miller, 1982 , 1986 ; Molholm et al, 2002 , 2006 ; see, however, Otto 
& Mamas sian, 2012 , for a challenge to this standard interpretation). 

1 .3 The "rules" of multisensory integration: Spatial, temporal, and intensive 

Unisensory stimuli that are closely aligned in space (the "spatial rule") and time (the "temporal rule") 
are more likely to produce multisensory response facilitation than are stimuli that are temporally and/ 
or spatially disparate, where the latter may even result in response suppression (Holmes & Spence, 
2005 ; Meredith & Airman, 2009 ; Meredith & Stein, 1986 ; Meredith, Nemitz, & Stein, 1987 ). Response 
facilitation may be especially robust for weak (i.e. near threshold) unisensory stimuli (the "inverse 
effectiveness rule"). Inverse effectiveness is most conspicuous in the responses of multisensory neu- 
rons in the superior colliculus (Meredith & Stein, 1986 ). 

1.3.1 The inverse effectiveness rule 

Results from both behavioral and electrophysiological studies have been interpreted to support the 
inverse effectiveness rule (Callan, Callan, Kroos, & Vatikiotis-Bateson, 2001 ; Diederich & Colonius, 
2004 ; Frassinetti, Bolognini, & Ladavas, 2002 ; Lakatos, Chen, O'Connell, Mills, & Schroeder, 2007 ; 
Serino, Fame, Rinaldesi, Haggard, & Ladavas, 2007 ; Senkowski, Saint- Amour, Hofle, & Foxe, 2011 ). 
The generality of the inverse effectiveness rule has recently been questioned on the grounds that pre- 
vious studies sampled an insufficient or incorrect range of stimulus intensities to critically test this 
hypothesis (Holmes, 2007 , 2009 ; Leavitt, Javitt, & Foxe, 2007; Ross, Saint- Amour, Leavitt, Javitt, 
& Foxe, 2007 ). Studies employing a range of stimulus intensities, whose results have been described 
as "generally consistent" with the inverse effectiveness rule (Lakatos et al., 2007 ), report maximal 
multisensory facilitation at intermediate (not the lowest) levels of stimulus intensity. Other studies 
(Senkowski et al., 2011 ), while testing a wide range of stimulus intensities, may still not have sampled 
at the lowest end of the perceptible intensity continuum. 



CP(RT < t\AV) < CP(RT < /-SOA AV |A) + CP(RT < t\ V), 



(1) 



where A is delayed relative to V, and 



CP(RT < t\PN) < CP(RT < t\A) + CP(RT < /-SOA AV | V), 



(2) 
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1.3.2 The temporal rule: Physiological simultaneity 

Stimuli are intrinsically processed at different rates by different sensory modalities. For example, 
human auditory evoked cortical potentials onset as early as 10 ms (Celesia & Puletti, 1971 ) whereas the 
CI component of the visual evoked potential, which reflects activity in early retinotopically mapped 
visual areas (VI /V2), onsets much later at 45-60 ms (Clark & Hillyard, 1996 ; Foxe & Simpson, 2002 ; 
Foxe et al., 2008 ; Jeffreys & Axford, 1972 ; Murray et al., 2001 ). In addition to processing speed dif- 
ferences between sensory channels due to intrinsic factors such as unequal pathway length, dissimilar 
axonal conduction velocity and/or variations in synaptic complexity, there is the variable influence 
of stimulus intensity. Simple RT is inversely related to stimulus intensity, a phenomenon known as 
Pieron's law (Jaskowski, 1985 ; Mansfield, 1973 ; Pieron, 1952 ; Prestrude, 1971 ; Roufs, 1963 ), and 
intensity-dependent variations in the latency of neural activity occur in the primate visual system 
(Barlow, Snodderly, & Swadlow, 1978 ; Maunsell et al., 1999 ) and in the auditory system of both cat 
(Eggermont, 1998 ; Phillips, 1998 ) and human (Stufflebeam, Poeppel, Rowley, & Roberts, 1998 ). 

There is general agreement that multisensory facilitation requires that the neural activity evoked 
by unisensory stimuli converge synchronously onto multisensory coincidence detectors — that is, that 
the unisensory inputs must exhibit physiological simultaneity (Hershenson, 1962 ; Miller, 1986 ; Raab, 
1962 ). A direct example is that multisensory neurons in the cat's superior colliculus exhibit optimal 
multisensory integration when the activity elicited by unisensory stimulation occurs at roughly the 
same post-stimulus latency (Stein & Meredith, 1993 ). Likewise, human behavioral data indirectly sug- 
gest that multisensory facilitation of simple RT occurs at stimulus onset asynchronies (SOAs) corre- 
sponding to the difference in simple RT to the unisensory stimuli, where this RT difference is presumed 
to reflect the difference in intrinsic processing speed (Diederich & Colonius, 2004 ; Hershenson, 1962 ; 
Miller, 1986 ). If optimal multisensory facilitation requires the synchronous convergence of unisensory 
signals onto a multisensory coincidence detector, and if the post-stimulus latency of evoked activity 
depends on stimulus intensity, then changes in the relative intensities of the component unisensory 
visual and auditory stimuli should cause systematic changes in the SOA AV at which optimal multisen- 
sory facilitation (neural coactivation) occurs. 

1 .4 The present experiment 

We measure how the optimal SOA AV for multisensory integration (as indexed by violations of the race 
model) depends on both absolute and relative unisensory stimulus intensities. An evaluation of the 
dependence of optimal SOA AV on absolute stimulus intensity addresses the validity and generality of 
the inverse effectiveness rule, while its dependence on relative stimulus intensity tests the predictions 
of the strong version of the physiological simultaneity hypothesis. 

2 Condition 1 

Condition 1 measured the range of SOA AV over which neural coactivation occurs for two relatively 
weak unisensory stimuli. 

2.1 Method 

2.1.1 Participants 

Participants (n = 4; two male; mean age = 31 years) possessed normal (or corrected to normal) vision 
and normal hearing. All experiments were conducted in accordance with the Code of Ethics of the 
World Medical Association (Declaration of Helsinki) for experiments involving humans. Prior to their 
participation in the study, all participants provided written informed consent. All procedures were 
approved by the institutional review board of North Dakota State University. 

2. 1.2 Stimuli and apparatus 

Visual stimuli were circular Gabor patches which, when viewed from 114 cm, possessed a spatial 
frequency of 1 cycles/degree and a Gaussian envelope with a standard deviation of 1°. Gabor patches 
were centered at 2.25° eccentricity from fixation in the upper left visual quadrant ( Figure 1 ). Stimuli 
were presented on a CRT (mean luminance = 60 cd/m 2 ; monitor refresh rate = 100 Hz). Gabor 
contrast for individual participants ranged between 1% and 4% across all conditions (see individual 
conditions for details). For condition 1, Gabor contrast ranged from 1.0% to 1.7%. Visual stimulus 
duration was 100 ms. 
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Figure 1. Visual stimuli were circular Gabor patches (l%-4% contrast) which possessed a spatial frequency of 
1 cycle/degree and a Gaussian envelope with a standard deviation of 1°. Gabor patches were centered at 2.25° 
eccentricity from fixation (cross) in the upper left visual quadrant. Scale bar =1°. 



The auditory stimulus was a 1-KHz pure tone of variable loudness (range = 31.1-49.0 dB A ) pre- 
sented via a speaker approximately co-located with the visual stimuli. Auditory stimulus duration was 
100 ms. 

2.1.3 Procedure 

2.1.3.1 Pretest. Prior to the experiment, all participants completed a pretest designed to equate 
the detectability of the unisensory stimuli. The pretest paradigm was a single- interval go/no-go signal 
detection task. Participants responded via button press as quickly and accurately as possible to the 
detection of any A or V stimulus. Participants performed 15 blocks of trials for a total of 30 trials per 
stimulus condition. Each block consisted of a total of 50 trials: 24 unisensory A stimuli (2 X 12 levels 
of dB attenuation), 24 unisensory V stimuli (2X12 contrasts), and 2 catch (no-signal) trials. 

Sensitivity {d') was calculated according to the equation d' = Z H — Z FA , where Z H denotes the 
Z- transformed hit rate (hits/signal trials) and Z FA denotes the Z- transformed false-alarm rate (false 
alarms/no-signal trials). Nonlinear least-squares regression to a logistic function interpolated stimulus 
intensities yielding criterion performance (d f ~ 2). 

2.1.3.2 Experiment. A single-interval go/no-go signal detection task was employed. Trials 
commenced with the appearance of a fixation cross. After a variable interval (1,000-2,000 ms), the 
first stimulus (SI) was presented. The second stimulus (S2) was presented following a variable SO A. 
Participants responded via button press as quickly and accurately as possible to the detection of any 
stimulus. RT was recorded to the nearest millisecond. Trials terminated after subject response or after 
1,500 ms. Figure 2 illustrates the sequence of events in multisensory trials. 

Participants completed a total of 34 blocks of 75 trials each, for a grand total of 2,550 trials. Each 
block included 9 V trials, 9 A trials, 9 no-stimulus catch trials, and 3 AV trials at each of 16 SOAs 
ranging (in 20-ms increments) from — 100 ms (A— >V) to 200 ms (V— >A). Each subject's RT distribu- 
tion was trimmed to exclude outliers (e.g., 100 ms > RT > 1,000 ms). The trimmed RT distributions 
possessed nearly equal numbers of trials contributed by each subject and subsequent analyses in all 
experimental conditions were conducted on RT distributions pooled across participants. 

The intensities of the unisensory stimuli were adjusted (as necessary) after every 1 8 trials to ensure 
a mean sensitivity of d' ~ 2, which corresponds to an 87% correct response rate in a two- alternative 
forced-choice task, where 75% correct is typically taken as threshold performance. Thus, stimuli in con- 
dition 1 were sufficiently strong that participants could reliably detect them, while sufficiently weak that 
performance in this condition could be meaningfully contrasted to that observed in subsequent experi- 
mental conditions where stimuli were highly suprathreshold (d' > 4; 99.9% correct response rate). 
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Figure 2. Multisensory trials commenced with the appearance of a fixation cross. After a variable interval 
(1,000-2,000 ms) the first stimulus (SI) was presented. The second stimulus (S2) was presented following a 
variable SOA. Participants responded via button press as quickly and accurately as possible to the detection of any 
stimulus. Response time (RT) was recorded to the nearest millisecond. Trials terminated after subject response 
or after 1,500 ms. 

2.1.4 Data analysis 

2. 1 .4. 1 Multisensory response facilitation/inhibition. To test whether RTs to multisensory stimuli 
were significantly faster (or slower) than RTs to unisensory stimuli, mean RT in all 1 6 SOA AV conditions 
was compared with the fastest (and slowest) mean unisensory RTs (A or V) using independent samples 
t- tests. 

2.1.4.2 Multisensory sensitivity enhancement. Mean sensitivity (d') at each SOA AV was 
calculated using a bootstrapping procedure (Foster & Bischof, 1991 ). Response distributions for each 
SOA AV condition (hits) were combined with an equal size sample taken at random from the no-signal 
condition (false alarms). The distribution of hits and false alarms was exhaustively sampled (with 
replacement, 1 ,000 iterations) to generate sampling distributions of d' from which means and standard 
errors were obtained. 

2.1 .4.3 Neural coactivation (Miller's inequality analysis). RT data were trimmed (100 ms > RT 
> 1 ,000 ms) and cumulative distribution functions (CDFs) were created for each stimulus condition 
(A, V, AV). Sampling distributions were bootstrapped by exhaustively resampling each CDF (with 
replacement, 1,000 iterations). A sampling distribution of Miller's inequality values was generated 
by subtracting the multisensory CDF predicted by probability summation, AV P = (A + V), from the 
CDF observed for each multisensory condition at each iteration such that MI f = AV — AV P , where MI f 
indicates facilitative MI. This process yielded a mean value (and standard error) of Miller's inequality 
for each SOA AV . Figures 8(c) and (d), condition 4, provide a graphic illustration of CDF comparisons 
between unisensory and simultaneous multisensory conditions. 

2.2 Results and discussion 

2.2.1 Multisensory response facilitation/inhibition 

Mean RTs for condition 1 are plotted as a function of SOA AV in Figure 3(a) . While possessing equally 
detectable stimulus intensities (d' ~ 2), mean RT to the unisensory A stimulus was nevertheless faster 
(424.6 ms) than to the unisensory V stimulus (431.9 ms). This difference in RT did not reach sig- 
nificance (t im = - 1.206, p = 0.228) and likely reflects the shorter latency of auditory versus visual 
cortical responses (Celesia & Puletti, 1971 ; Jeffreys & Axford, 1972 ). To assess whether multisensory 
stimulation produced response facilitation, mean RT in each AV condition was compared with that in 
the fastest unisensory condition (A). Green bars identify those AV SO As where results of independ- 
ent samples Mests indicated that significant multisensory response enhancement occurred. Significant 
response facilitation occurred at SOA AV values of —60, —40, —20, 0, 20, and 40 ms (t l303 = —2.891, 
p = 0.004; t m3 = -2.944, p = 0.003; t l295 = -3.560,/? < 0.001; t ms = -4.620, p < 0.001; t l303 = 
—2.933,/? = 0.003; and t l290 = —1.957,/? = 0.051; respectively). To assess whether multisensory 
stimulation might produce response inhibition, mean RT in AV conditions was compared with that in 
the slowest unisensory condition (V). Red bars indicate that significant response inhibition occurred at 
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(b) Condition 2: A (d' * 2); V (d' > 4) 
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Figure 3. Mean RTs in the four experimental conditions plotted as a function of SOA AV . Mean RT in each AV 
condition was compared with that in the fastest unisensory condition (A or V). Green bars identify those AV 
SOAs where results of independent samples f-tests indicated that significant multisensory response enhancement 
occurred. Mean RT in each AV condition was also compared with that in the slowest unisensory condition (A or 
V). Red bars identify those AV SOAs for which significant multisensory response inhibition occurred. 



SOA AV of 140, 160, 180, and 200 ms (t U93 = 2.712,/? = 0.007; t U9S = 2.626,/? = 0.009; t n95 = 2.141, 
p = 0.032; and/ 1298 = 4.100,/? < 0.001; respectively). 

2.2.2 Multisensory sensitivity enhancement 

If sensitivities to the unisensory A and V stimuli combine probabilistically then the sensitivity in 
multisensory (AV) conditions (d' AVp ) should equal the quadratic sum of the unisensory sensitivities 
(Campbell & Green, 1965 ; Legge, 1984 ): 

d avp = V(^o) 2 + TO" 2 - (3) 

Figure 4 plots the mean bootstrapped values of observed sensitivity (d' AWo , ±2 and ±3 SEM) in all 16 
multisensory conditions as a function of SOA AV . The average observed unisensory sensitivities were 
d' Ao = 1.911, and d' Yo = 1.910, and the multisensory sensitivity predicted by probability summation 
(^av p = 2.699) is plotted as a horizontal line. Although there is a significant multisensory facilitation 
of mean RT, sensitivity to multisensory stimuli does not differ significantly (p > 0.01) from that pre- 
dicted by probability summation at any SOA AV . 

2.2.3 Neural coactivation (Miller's inequality analysis) 

Figure 5(a) spectrum codes the value of Miller's inequality as a function of RT and SOA AV . Figure 5(b) 
is a magnified view of the region exhibiting significant (/? < 0.05) violations of the race model (posi- 
tive values). Noteworthy is that positive values of the inequality occur exclusively at an SOA AV value 
of 0 ms (physical simultaneity), across a range of RT (200-325 ms) with a peak value (0.0360) occur- 
ring at 292 ms ( Figure 5c ). 

3 Condition 2 

Condition 2 was designed to reveal how changing the intensity of the V stimulus might influence its 
integration with the relatively weak A stimulus. Specifically, we tested whether increasing the con- 
trast of the V stimulus caused a change in the optimal SOA AV for multisensory integration relative to 
condition 1. If physiological simultaneity determines multisensory facilitation, then increasing the 
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Figure 4. Mean bootstrapped values of observed sensitivity (d' AYo , ±2 and ±3 SEM) in all 16 multisensory 
conditions as a function of SOA AV . The average observed unisensory sensitivities were d' Ao = 1.911 and 
dy 0 = 1.910, and the multisensory sensitivity predicted by probability summation (d' AYp = 2.699) is plotted as a 
horizontal line. Although there is a significant multisensory facilitation of mean RT, sensitivity to multisensory 
stimuli does not differ significantly (p > 0.01) from that predicted by probability summation at any SOA AV . 
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Figure 5. Test of the race model for condition 1 . Panel (a) shows spectrum-coded mean values of Miller 's inequality 
as a function of RT and SOA AV . Panel (b) shows a magnified view of the region exhibiting a significant (p < 0.05) 
violation of the race model that occurred exclusively at an SOA AV of 0 ms. Panel (c) plots the mean value (thick 
line) and 95% confidence intervals (thin lines) for Miller's inequality as a function of RT for simultaneous SOA AV . 
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intensity of the V stimulus (which increases the speed of visual processing and reduces mean RT to V 
stimuli) should shift the optimal SOA AV to more negative values such that, relative to condition 1 , the 
A stimulus will now need to be presented earlier in time with respect to the V stimulus. Conversely, the 
optimal SOA AV will remain unchanged if physical simultaneity is critical for multisensory integration. 

3.1 Method 

The participants in condition 2 were the same as in condition 1 . The experimental methods and pro- 
cedures were as in condition 1 except that V stimulus contrast was increased to levels producing a 
criterion sensitivity of d' > 4 for each participant (contrast range = 3.4%-4.0%). To ensure this cri- 
terion sensitivity, d' was calculated for unisensory A and V stimuli after every 1 8 trials throughout the 
experiment and V stimulus contrast and A stimulus intensity were adjusted as necessary. 

3.2 Results and discussion 

3.2.1 Multisensory response facilitation/inhibition 

Mean RTs in condition 2 are plotted as a function of SOA AV in Figure 3(b) . As expected, mean RT 
to the unisensory V stimulus was now significantly faster (315.8 ms) than to the unisensory A stimu- 
lus (450.0 ms) (t l9S4 = —25.158,/? < 0.001). This highly significant reversal in the relative speed 
of responses to A and V stimuli reflects the approximately threefold increase in V stimulus contrast 
relative to condition 1, and is an illustration of Pieron's law. To assess whether multisensory stimula- 
tion produced response facilitation, mean RT in AV conditions was compared with that in the fastest 
unisensory condition (V). The green bars indicate that significant response facilitation occurred at 
SOA AV values of 0 and 20 ms (t l597 = - -2.272, p = 0.023 and t l597 = — 2.160, p = 0.031, respectively). 
There was no multisensory response inhibition at any SOA AV . 

3.2.2 Neural coactivation (Miller's inequality analysis) 

Figure 6(a) spectrum codes the values of Miller's inequality as a function of RT and SOA AV for condi- 
tion 2. A magnified view of the region exhibiting significant (p < 0.05) violations of the race model is 
shown in Figure 6(b) . As in condition 1, significant violations of Miller's inequality occurred for phys- 
ically simultaneous A and V stimuli, as well as at an SOA AV of 20 ms. At an SOA AV of 0 ms ( Figure 6c ), 
violations occurred for physically simultaneous stimuli across a range of RT (217-290 ms), with a 
peak at 244 ms. The fact that the greatest violations of Miller's inequality occur when unisensory 
stimuli are physically simultaneous is clearly incompatible with the strong version of the physiologi- 
cal simultaneity hypothesis, which posits that neural coactivation should occur at an SOA that closely 
corresponds to the difference in mean RT to the unisensory stimuli. The results of condition 2 also 
clearly show that neural coactivation can occur when a more rapidly processed stimulus (V) precedes 
a more slowly processed stimulus (A). The results of condition 2 are also inconsistent with the inverse 
effectiveness rule, because increasing the intensity of the V stimulus actually increased the magnitude 
of the violation of Miller's inequality (0.0360 vs. 0.0530 in conditions 1 and 2, respectively). 

4 Condition 3 

Condition 3 is complimentary to condition 2 and was designed to reveal how changing the intensity of 
the A stimulus might influence its integration with a relatively weak V stimulus. Specifically, we tested 
whether increasing the intensity (loudness) of the A stimulus caused a change in the optimal SOA AV for 
multisensory integration relative to condition 1 . Again, if physiological simultaneity is necessary for 
multisensory facilitation, then increasing the intensity of the A stimulus (which will increase the speed 
of auditory processing and reduce mean RT to A stimuli) should shift the optimal SOA AV for neural 
coactivation to more positive values such that, relative to condition 1 , the V stimulus would need to 
be presented earlier with respect to the A stimulus. Conversely, the optimal SOA AV will not change if 
physical simultaneity determines multisensory integration. 

4.1 Method 

The participants in condition 3 were the same as in conditions 1 and 2. The experimental methods and 
procedures were as in condition 1 , except that A stimulus intensity was increased to levels producing a 
criterion sensitivity d' > 4 (intensity range = 44.5^9.0 dB A ). In order to ensure criterion sensitivity, 
d' was calculated for unisensory A and V stimuli after every 1 8 trials throughout the experiment and V 
stimulus contrast and A stimulus intensity were adjusted as necessary. 
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Figure 6. Test of the race model for condition 2. Panel (a) shows spectrum-coded mean values of Miller 's inequality 
as a function of RT and SOA AV . Panel (b) shows a magnified view of the region exhibiting significant (p < 0.05) 
violations of the race model that occurred at SOA AV values of 0 and 20 ms. Panel (c) plots the mean value (thick 
line) and 95% confidence intervals (thin lines) for Miller's inequality as a function of RT for simultaneous SOA AV . 



4.2 Results and discussion 

4.2.1 Multisensory response facilitation/inhibition 

Mean RTs in condition 3 are plotted as a function of SOA AV in Figure 3(c) . As expected, mean RT to 
the unisensory A stimulus (301.5 ms) was significantly faster than that to the unisensory V stimulus 
(441.7 ms) (t l97S = —32.247, p < 0.001). The significantly faster mean response to the intense A 
stimulus in condition 3, relative to conditions 1 and 2, is another illustration of Pieron's law. To assess 
whether multisensory stimulation produced response facilitation, mean RT in AV conditions was com- 
pared with that in the fastest unisensory condition (A). Green bars indicate that significant response 
facilitation occurred at SOA AV values of —20 and 0 ms (t m9 = —2.340,/? = 0.019; t m6 = —4.145, 
p < 0.001, respectively). There was no multisensory response inhibition at any SOA AV . 

4.2.2 Neural coactivation (Miller's inequality analysis) 

Figure 7(a) spectrum codes the values of Miller's inequality as a function of RT and SOA AV for condi- 
tion 3. A magnified view of the region exhibiting significant violations of the race model is shown in 
Figure 7(b) . At an SOA AV of 0 ms ( Figure 7c ), positive values of Miller's inequality occur across a 
range of RT (192-330 ms), with a peak at 288 ms. As in conditions 1 and 2, significant violations of 
Miller's inequality occurred for physically simultaneous A and V stimuli, as well as at an SOA AV of 
—20 and —40 ms. Once again, the fact that violations of Miller's inequality have the greatest mag- 
nitude when unisensory stimuli are physically simultaneous is clearly incompatible with the strong 
version of the physiological simultaneity hypothesis, and the results of condition 3 confirm that neu- 
ral coactivation can occur when a more rapidly processed stimulus (A) precedes the more slowly 
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Figure 7. Test of the race model for condition 3. Panel (a) shows spectrum-coded mean values of Miller's 
inequality as a function of RT and SOA AV . Panel (b) shows a magnified view of the region exhibiting significant 
(p < 0.05) violations of the race model, which occurred at SOA AV values of —40, —20, and 0 ms. Panel (c) plots 
the mean value (thick line) and 95% confidence intervals (thin lines) for Miller's inequality as a function of RT 
for simultaneous SOA AV . 



processed stimulus (V). The results of condition 3 are also inconsistent with the inverse effectiveness 
rule, because increasing the intensity of the A stimulus significantly increased the magnitude of the 
violation of Miller's inequality (0.0360 vs. 0.1113 in conditions 1 and 3, respectively). 

5 Condition 4 

Finally, condition 4 tested whether increasing the intensities of both A and V stimuli to clearly suprath- 
reshold levels (d' > 4) would result in a pattern of optimal SOA AV for multisensory integration that 
resembled that of condition 1 . 

5.1 Method 

The participants in condition 4 were the same as in condition 1 . The experimental paradigm replicated 
condition 1 with the following exception. Both V and A stimuli were increased in intensity to levels 
producing criterion sensitivities of d' > 4. 

5.2 Results and discussion 

5.2.1 Multisensory response facilitation/inhibition 

Mean RTs in condition 4 are plotted as a function of SOA AV in Figure 3(d) . As in condition 1 , mean 
RT to the unisensory A stimulus was significantly faster (286.3 ms) than that to the unisensory V 
stimulus (316.7 ms) (t 2402 — —8.74,/? < 0.01). The significantly faster mean responses to the highly 
suprathreshold A and V stimuli in condition 4, relative to condition 1 (t 2Ul = —28.338,/? < 0.01 and 
t 2U2 = —26.030,/? < 0.01, respectively), again illustrate Pieron's law. To assess whether multisensory 
stimulation produced response facilitation, mean RT in AV conditions was compared with that in the 
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fastest unisensory condition (A). The green bars indicate that significant response facilitation occurred 
at SOA AV values of -80, -20, and 0 ms (t m3 = -2.03, p = 0.043; f 1613 = -2.67, p = 0.008; and 
t mo = —5.72,/? < 0.001, respectively). There was no multisensory response inhibition at any SOA AV . 

5.2.2 Neural coactivation (Miller's inequality analysis) 

Figure 8(a) spectrum codes the values of Miller's inequality as a function of RT and SOA AV for condi- 
tion 4. A magnified view of the region exhibiting significant violations of the race model is shown in 
Figure 8(b) . As in condition 3, positive values of Miller's inequality occurred for SOA AV —20 and 0 
ms, a result that is incompatible with the strong version of the physiological simultaneity hypothesis. 
At an SOA AV of 0 ms, positive values of Miller's inequality occur across a range of RT (171-257 ms), 
with a peak at 232 ms. The results of condition 4 are directly contradictory to the inverse effectiveness 
rule, because increasing the intensity of both unisensory stimuli significantly increased the magnitude 
of the violation of Miller's inequality (0.0360 vs. 0.1678 in conditions 1 and 4, respectively). 

Figures 8(c) and (d) illustrate how the RT data were analyzed in order to compute values of 
Miller's inequality and establish confidence intervals. CP distributions as a function of RT were con- 
structed for the A (purple) and V (blue) unisensory stimulus conditions, and for their multisensory 
combination, AV 0 , at all values of SOA AV (SOA AV = 0 ms is shown in red). The sum of the CP distribu- 
tions for the unisensory conditions (A + V) is the CP predicted by the race model (black). The value 
of Miller's inequality is computed at each value of RT by subtraction: CP(AV 0 ) — CP(A + V). When 
the CP of observed RT exceeds that predicted by the race model, the value of Miller's inequality is 
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Figure 8. Test of the race model for condition 4. Panel (a) shows spectrum-coded mean values of Miller's 
inequality as a function of RT and SOA AV . Panel (b) shows a magnified view of the region exhibiting significant 
(p < 0.05) violations of the race model, which occurred at SOA AV values of -20 and 0 ms. Panel (c) plots the 
mean value (thick line) and 95% confidence intervals (thin lines) for Miller's inequality as a function of RT 
for simultaneous SOA AV . Panel (c) shows the CP distributions for the two unisensory conditions (A: purple; V: 
blue) and their sum (A + V: black), which is the CP distribution predicted by the race model. The CP distribution 
observed for AV stimulation at an SOA AV of 0 ms (AV 0 ) is shown in red. Panel (d) plots the mean value (thick 
line) and 95% confidence intervals (thin lines) for Miller's inequality as a function of RT for simultaneous SOA AV . 
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positive. These violations are represented by the segment of the AV 0 CP distribution that lies above 
the (A + V) CP distribution in Figure 8(c) . Miller's inequality is plotted (thick line) as a function of 
RT (from 150 to 350 ms) in Figure 8(d) . Confidence intervals were derived by a bootstrapping proce- 
dure (Foster & Bischof, 1991 ) where the AV 0 and A + V CP distributions were sampled exhaustively 
with replacement to generate a sampling distribution of Miller's inequality (N = 1,000). Thin lines in 
Figures 5-7(c) and 8(d) plot 95% confidence intervals for the value of Miller's inequality. 

6 General discussion 

Taken together, the results of the four experimental conditions reveal that audiovisual multisensory 
facilitation, as indexed by significant violations of the race model, occurs only over a narrow range of 
stimulus onset asynchronies which invariably includes physical simultaneity. Manipulations of stimu- 
lus intensity that changed the speed of unisensory processing, as revealed by significant alterations 
of mean RT, had no influence on the range of SOA AV over which multisensory interaction occurred. 
Moreover, when the range of SOA AV over which significant violations of the race model occurred did 
extend beyond physical simultaneity (condition 2: 20 ms; condition 3: —20 and —40 ms; condition 4: 
—20 ms), the extension was such that the more rapidly processed stimulus needed to precede the more 
slowly processed stimulus, a result that is, in fact, opposite to the strong version of the physiological 
simultaneity hypothesis (Hershenson, 1962 ), which predicts that the more slowly processed stimulus 
should need a "head start" in order to arrive at some central site simultaneously with a more rapidly 
processed stimulus. It should be noted that violations of Miller's inequality at all non-zero SO As were 
smaller than those at simultaneity. Finally, there was a surprising lack of evidence for multisensory 
facilitation with respect to sensitivity (d'). 

6.1 Inverse effectiveness 

Although the rule of inverse effectiveness has sometimes been upheld as a universally observed char- 
acteristic of multisensory integration (Lakatos et al., 2007 ; Meredith & Stein, 1986 ), a number of 
studies have reported findings that are inconsistent with this rule (Lakatos et al., 2007 ; Ross et al., 
2007 ). Our results likewise do not support the inverse effectiveness rule because the magnitude of the 
violations of Miller's inequality we observed generally increased with increasing stimulus intensity 
(at SOA AV = 0 ms: 0.0360, 0.0530, 0.1113, and 0.1678 in conditions 1-4, respectively). Our results 
are, however, consistent with a reinterpretation (Holmes, 2007 ) of the results of Alvarado, Vaughan, 
Stanford, and Stein ( 2007 ), who showed that whereas multisensory enhancement in neurons in the 
cat's superior colliculus obeyed the rule of inverse effectiveness when analyzed in terms of relative 
spike rate increase, they demonstrated the behavior we observe, viz., increasing effectiveness with 
increasing stimulus intensity, when analyzed in terms of absolute spike rates. On the contrary, the 
range of SOA AV over which statistically significant decreases in simple mean RT occur is wider for the 
weakest stimuli (condition 1) than for the other conditions ( Figure 3 ), but increases in the time window 
of integration is not what is typically meant by inverse effectiveness. 

6.2 Intensity-adjusted latency coding 

Although physiological activity resulting from two unisensory signals must simultaneously converge 
on a multisensory "coincidence detector" for facilitative MI to occur, we find that such facilitation 
occurs nearly exclusively for physically simultaneous multisensory occurrences, independent of fac- 
tors that differentially affect unisensory processing time, such as stimulus intensity. This makes sense 
from an ecological perspective, for if multisensory facilitation has aided survival, the advantage it 
confers must be the enhanced processing of genuine physical events (e.g., the sights and sounds of 
predators or prey which, because they have a common cause, are physically simultaneous), not merely 
to physiological simultaneities some (potentially large) fraction of which are accidental. Although 
the evolutionary advantage of this result is undeniable, the mechanism whereby it is accomplished is 
unclear. 

Consider the case in which facilitative MI occurs at a coincidence detector where the response 
latencies for a given pair of simultaneously presented unisensory stimuli are exactly equal. Varying 
the relative intensities (and hence the processing speed) of the physically simultaneous unisensory 
stimuli will cause their physiological responses to convergence at this detector asynchronously, and 
yet we find that physically simultaneous stimuli integrate despite variations in intensity. Conversely, 
the physically asynchronous occurrence of two unisensory stimuli can, depending on their relative 
intensity, result in spuriously coincident physiological responses that nevertheless fail to result in 
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facilitation. This suggests that the time of arrival at multisensory sites must be corrected for latency 
differences caused by adventitious variations in stimulus intensity. In other words, the system must 
"take account" of differences in physiological response latency that are unrelated to true unisensory 
SO A in order to reject false correspondences. It is well known that neural networks can undergo Heb- 
bian learning, wherein the strength of synaptic connections is modified based on experience. The bio- 
physical bases of temporal coincidence detection could involve mechanisms of spike-time dependent 
synaptic plasticity (Song, Miller, & Abbott, 2000 ) and/or synaptic scaling (Turrigiano, Leslie, Desai, 
Rutherford, & Nelson, 1998 ). Our novel idea that synapses might additionally be "tuned" to particular 
stimulus intensities is purely speculative. In short, however, physiological response simultaneity is a 
necessary but not sufficient condition for facilitative MI. 

One possible mechanism for achieving this outcome is shown in Figure 9 . Assume a neural net- 
work of AV coincidence detectors (labeled 1-5 in Figure 9 ) which integrate unisensory inputs (i.e., 
fire) exclusively when A and V inputs arrive synchronously, thus reflecting the necessary condition of 
physiological simultaneity. The coincidence detectors receive input from both A and V sensory recep- 
tors via delay lines and are thus place coded (Jeffress, 1948 ). A given afferent A or V signal ultimately 
supplies input to the entire network of coincidence detectors at a continuum of latencies. In Jeffress' 
( 1948 ) theory of binaural sound localization, differences in the time of arrival of sound at the two 
ears (the interaural time difference) result in physiological convergence that varies in location within 
the network of coincidence detectors such that the spatial location of the sound source is read out by 
the relative spatial location of the activated coincidence detector. This explanation can be adapted to 
explain our results. 

We begin by assuming that organisms have ubiquitous access to the physiological activity gener- 
ated by genuinely simultaneous multisensory events in their environment, that is, that they have access 
to ground truth. Thus, over the course of their development organisms will accrue robust Bayesian 
priors with respect to multisensory convergence. The class of multisensory stimulation for which exact 
latency/intensity information is known results from self-generated events. For example, tapping an 
object with the hand (or a tool), or throwing a projectile whose impact with a nearby surface produces 
both visual and auditory consequences, etc., produces afferent visual, auditory, and haptic/kinesthetic 
signals of known common origin. Because these signals routinely vary in intensity, the latencies at 
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Figure 9. An illustration of how learned associations (Bayesian priors) between physical simultaneity and relative 
stimulus intensity might be represented in a place-coded delay line network of coincidence detectors. Panels (a-c) 
refer to situations where physically simultaneous A and V events vary in relative intensity. In panel (a), the A and 
V inputs are of nominally equivalent relative intensity (denoted by the similar size of the delay lines, lettering, 
and the circular input and synaptic symbols), and the afferent signals (coded red) converge synchronously on and 
activate coincidence detector 3 (red). In panels (b) and (c), simultaneous A and V events vary in intensity. The 
more intense stimulus (thick line) propagates through the network more rapidly than the less intense stimulus 
(thin line), and their signals thus converge on coincidence detectors 1 or 5. Panel (d) illustrates two inputs of 
nominal equivalent intensity where the visual stimulus is delayed in time of onset relative to the auditory stimulus 
(represented here as added pathway length). 
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which these signals converge at multisensory coincidence detectors can, over repeated stimulation, 
result in the accumulation of intensity-adjusted probability distributions that form the basis for com- 
puting the posterior probability that any set of incoming multisensory signals have a common origin. 

One way in which the Bayesian priors could be instantiated in a delay-line network is illustrated 
in Figure 9 . 

Figures 9(a-c) refer to situations where physically simultaneous A and V events vary in relative 
intensity. In Figure 9(a) , the A and V inputs are of nominally equivalent relative intensity (denoted 
by the similar size of the delay lines, lettering, and the circular input and synaptic symbols), and the 
afferent signals (coded red) converge synchronously on and activate coincidence detector 3 (red). No 
other coincidence detector receives simultaneous input. Figures 9(b) and (c) describe situations where 
simultaneous A and V events vary in intensity. The more intense stimuli (thick lines) propagate through 
the network more rapidly than the less intense stimuli (thin lines), and their signals thus converge on 
coincidence detector 1 or 5. Because we are assuming that in each case the physical origin of the 
afferent signals is known to have been common, physical stimulus synchrony can be disambiguated 
and correctly read out by the network only if the coincidence detectors are trained (learn) to fire when 
the constituent inputs possess the appropriate relative intensities. Thus, in Figure 9(d) , two inputs of 
nominal equivalent intensity are illustrated where the visual stimulus is delayed in time of onset rela- 
tive to the auditory stimulus (represented here as added pathway length). Being of nominal equivalent 
intensity, the two signals propagate through the network at similar speeds and converge on coincidence 
detector 4. However, because this coincidence detector has "learned" that physically simultaneous A 
and V events only converge at its location when they possess unequal intensities (intensity tuning is 
indicated by the sizes of the circular synaptic contacts), it rejects this physiological simultaneity as 
spurious and does not integrate. Animations demonstrating the four conditions of Figure 9 are avail- 
able as a supplementary flash file. 
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